Method and system of structural light-based 3d depth imaging using signal separation coding and error correction thereof

ABSTRACT

A 3D depth imaging method and system are disclosed. The 3D depth imaging method involves radiating light at a measurement target object using a projection means and imaging the light using an image receiving means, and includes the steps of assigning a unique transmitting side address to a signal corresponding to each pixel of the projection means to encode the signal; projecting multiple light patterns at the projection means to transmit the signal; receiving the encoded signal at the image receiving means; separating the received signal to restore the address; and determining a pixel position of the object using the transmitting side address and the restored address. With the 3D depth imaging method and system, it is possible to exactly separate signals received by the image receiving means even when the signals are overlap and the geometrical structure of the object varies, and it is also possible to obtain a depth image that is robust against ambient environmental noise.

BACKGROUND OF THE INVENTION

1. Field of the invention

The present invention relates to a 3D depth imaging method and system,and more particularly, to a 3D depth imaging method and system capableof more exactly imaging a 3D depth by exactly restoring a modifiedsignal using a new scheme called signal separation coding.

2. Description of the Prior Art

In general, a method of three-dimensional (3D) depth imaging using astructural light is recently receiving attention because it is suitablefor sensing a 3D environment in service robotics. The basic principle ofdepth imaging using a structural light, which is an active stereoscheme, is to radiate light at an object using a projection means suchas a projector, image the object irradiated with light using an imagereceiving means such as a camera, and observe the extent of distortionof the light due to the object in order to calculate the depth of theobject and obtain a depth image.

FIG. 1 is a schematic diagram illustrating a principle of a structurallight-based 3D depth imaging system. As shown in FIG. 1, a 3D positionof one point x on an object 100 is determined as an intersection pointof a straight line connected between an origin O_(p) of a projectionmeans and a point p on the retinal plane 200 of the projection means,and a straight line connected between an origin O of an image receivingmeans and a point q on a retinal plane 300 of the image receiving means.Accordingly, a depth image can be obtained by calculating thecoordinates of a point x as a pair of address values on each retinalplane at the points p and q after the projector and the camera arecalibrated. That is, the core of the depth imaging method employing sucha stereo scheme is to determine a pixel correspondence point between thereceived image and the projected image. With the determinedcorrespondence point, the depth can be easily calculated using simplegeometry.

For the accuracy of depth imaging, a light pattern projected by theprojection means is coded spatially and/or temporally according to atime sequence on a pixel array so that a spatial and/or temporal addressof the signal detected by the image receiving means uniquely determinesthe pixel correspondence point of the corresponding projection means.

Examples of such a conventional coding method include direct coding,spatial coding, temporal coding, and hybrid coding methods.

In the direct coding method, a grey and/or color level is directly usedin coding, and a disparity image is calculated through one patternframe. This method has the advantage of high speed. However, theaccuracy and robustness of the method are poor due to ambientillumination variation and noise.

In the spatial coding method, a specially designed, coded pattern, whichis arranged on a pixel array, is used. A De Brujin sequence, a quasirandom code or the like is also used. The thus coded pattern is used toprovide pixel address information from an adjacent pixel. Spatial codingobtains a disparity image from one or two frame patterns. This methodhas high speed and improved robustness in error correction because ofits address information. However, spatial coding is affected by signalmodification or complexity of an object because it uses spatiallyarranged address information for pixel correspondence.

Temporal coding uses coding patterns arranged along the time axis. Abinary code, an N-ary code, and a line shifting-based gray code havebeen suggested. Generally, temporal coding has higher accuracy thanspatial coding. This is because temporal coding has no order restrictionand is possible using a black and white color signal. However, temporalcoding is not suitable for a rapidly changing scene because of its useof a frame sequence.

Hybrid coding uses a mixture of temporal coding and spatial coding. Thiscoding method can obtain a robust depth image. However, hybrid codingcannot be used in a very complex environment because it has theshortcomings of spatial coding. Spatial coding and hybrid coding aresuitable for an object having a continuous surface. They may undergotransposition of the address sequence in a discontinuous surface.

Such conventional coding methods are focused on designing a spatialand/or time code in calculating a single pixel based on addressinformation. However, the conventional coding methods have a fundamentallimitation in calculating a higher-accuracy pixel correspondence pointneeded for exact depth imaging. In particular, they overlook a complexboundary surface such as occluding and shading the neighborhood of aboundary, or transposition that address information undergoes in pixelcorrespondence. Accordingly, the accuracy of depth imaging for a complexobject is very low. It results from the fact that the conventionalmethod is focused on coding the received signal based on addressinformation and does not consider a signal modification extent makingthe address information inexact at the image receiving means side.Further, even though a long time code sequence providing each pixel at aprojection means side having a unique address is used, it is noteffective in processing signal modification.

Further, it should be considered that the signal received by the imagereceiving means is affected by system/environmental noise such as lightscattering, reflectance change, and ambient illumination variation, andundergoes multiple code mixing in which the signal is mixed with pixelsadjacent to the projection means and even remote pixels. However,conventional methods overlook such matters.

Accordingly, the present inventors suggest a new coding method called a“signal separation scheme” to solve the aforementioned problems.

SUMMARY OF THE INVENTION

The present invention has been made to solve the aforementioned problemsof the conventional art. The present invention provides a structurallight-based 3D depth imaging method and system using a signal separatingand coding method capable of exactly separating an original signal evenwhen a plurality of signals received at an image receiving meansoverlap.

The present invention also provides a signal separating and codingmethod and system for correspondence point determination capable ofexactly separating a signal even when the signal is distorted by acomplex boundary surface of an object in 3D depth imaging.

The present invention also provides a signal separating and codingmethod and system for determining a correspondence point in structurallight-based 3D depth imaging having robustness against ambientsystem/environmental noise.

The present invention also provides a technique enabling easyinterpretation of change in the physical relationship between anartificial light source, an object, and a camera.

According to an aspect of the present invention, there is provided amethod of 3D depth imaging by radiating light at a measurement targetobject using a projection means and imaging the light using an imagereceiving means, the method comprising the steps of: assigning a uniquetransmitting side address to a signal corresponding to each pixel of theprojection means to encode the signal; projecting multiple lightpatterns at the projection means to transmit the signal; receiving theencoded signal at the image receiving means; separating the receivedsignal to restore the address; and determining a pixel position of theobject using the transmitting side address and the restored address.

The step of separating the received signal may comprise an errorcorrection step including the steps of calculating possible addresscandidate values corresponding to the transmitting side address; anddetermining an address according to a rule for obtaining an exactaddress from the candidate values, and the rule is selected from: a rulethat the restored address values gradually increase or decrease with nocontinuity when the signal is obtained from a slanted surface of theobject; a rule that the restored address values vanish or are transposedin part with no continuity when the signal is obtained from a cutsurface of the object; and a rule that the restored address valuesvanish with no continuity when the signal vanishes by a shadow of theobject.

The step of separating the received signal may comprise an errorcorrection step including the steps of: calculating possible candidatevalues of an address corresponding to the transmitting side address; anddefining an evaluation function by considering a group of addresscandidates at one's position and address candidates at neighboringpositions among the candidates, and determining the address using asearching method for maximizing or minimizing the evaluation function.

The step of separating the received signal may comprise an errorcorrection step including the steps of: calculating possible candidatevalues of an address corresponding to the transmitting side address; andcalculating an address value at a position having high reliability,fixing the address value at the position, and then determining thecandidates from address values in other areas in order to select theaddress from a plurality of candidate groups consisting of thecandidates.

The light pattern may be a binary light pattern, a color light pattern,a light pattern having a brightness value of a gray scale, a lightpattern of an ultraviolet or infrared area in a non-visible area, or amixture of these.

The step of encoding the signal may comprise using orthogonal signalsamong adjacent signals at the projection means.

All the orthogonal signals may be classified into one or morehierarchical signals, and use codes that are orthogonal to one anotherin each layer.

All the orthogonal signals may be classified into one or morehierarchical signals, and use a mixture of codes that are orthogonal toone another in each layer and codes that are not orthogonal to oneanother.

The step of encoding the signal may comprise using pseudo orthogonalsignals having orthogonality among adjacent signals of the projectionmeans.

The step of encoding the signal may comprise using statisticallyindependent signals among adjacent signals of the projection means.

According to another aspect of the present invention, there is provideda method of 3D depth imaging in a system including a projector and acamera, the method comprising the steps of: emitting a pattern from theprojector to an object; receiving the pattern using the camera;calculating a pixel correspondence relationship between the projectorand the camera through signal separation on each epipolar line; andcreating a new image at a time point of the projector after thecalculating step is performed.

According to still another aspect of the present invention, there isprovided a method for restoring a stereo image in a system including aprojector and a camera, the method comprising the steps of: emitting apattern from the projector to an object; receiving the pattern using thecamera; calculating a pixel correspondence relationship between theprojector and the camera through signal separation on each epipolarline; and creating a new image at a time point of the projector afterthe calculating step is performed, in which the stereo image is restoredusing an image obtained by the camera and the new image created by theprojector.

According to yet another aspect of the present invention, there isprovided a method for restoring a measuring depth in a system includinga projector and a camera, wherein the method restores the measuringdepth using both a structural light-based 3D depth imaging methodcomprising the steps: assigning a unique transmitting side address to asignal corresponding to each pixel of the camera to encode the signal;projecting multiple patterns from the projector to transmit the signal;receiving the encoded signal at the camera; separating the receivedsignal to restore the address; and determining a pixel position of anobject using the transmitting side address and the restored address, anda stereo image restoring method comprising the steps of: emitting apattern from the projector to the object; receiving the pattern usingthe camera; calculating a pixel correspondence relationship between theprojector and the camera through signal separation on each epipolarline; and creating a new image at a time point of the projector afterthe calculating step is performed.

According to yet another aspect of the present invention, there isprovided a structural light-based 3D depth imaging system comprising:projection means for radiating light at a measurement target object;image receiving means for imaging the light radiated from the projectionmeans; and processing means for imaging a 3D depth of the measurementtarget object, the processing means sequentially performing: encoding asignal corresponding to each pixel of the projection means by assigninga unique transmitting side address to the signal; projecting multiplelight patterns at the projection means to transmit the signal; receivingthe encoded signal at the image receiving means; separating the receivedsignal to restore an address; and determining a pixel position of theobject using the transmitting side address and the restored address.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features and advantages of the presentinvention will be more apparent from the following detailed descriptiontaken in conjunction with the accompanying drawings, in which:

FIG. 1 is a conceptual diagram illustrating a 3D imaging method using astructural light;

FIG. 2 is a conceptual diagram illustrating signal mixture according toa geometrical relationship of an object;

FIG. 3 is a flowchart illustrating a 3D imaging method using signalseparation according to an embodiment of the present invention;

FIG. 4 is a conceptual diagram illustrating a hierarchical orthogonalcode, one example of a code that can be used for signal separationcoding according to an embodiment of the present invention;

FIG. 5 is a conceptual diagram illustrating a process of separating ahierarchical orthogonal code in signal separation coding according to anembodiment of the present invention;

FIGS. 6 and 7 illustrate a signal separation map obtained by performingsignal separation on a specific epipolar line and a dual photographyimage created using the signal separation map; and

FIGS. 8 and 9 illustrate dual photography using signal separationcoding.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Hereinafter, preferred embodiments of the present invention will bedescribed with reference to the accompanying drawings. In the followingdescription and drawings, the same reference numerals are used todesignate the same or similar components, and such components will onlybe described once.

First, the concept of the present invention will be described.

In a 3D depth imaging method according to the present invention, it isimportant to determine an exact correspondence point between a pixelposition at a projection means such as a projector and a pixel positionat an image receiving means such as a camera in order to obtain an exactdepth image. From the determined correspondence point, 3D data of anobject is easily calculated using simple geometry. Because thecorrespondence point on a stereo image exists on an epipolar line 320shown in FIG. 1, it suffices to search only on the epipolar line. Inparticular, in a parallel stereo system in which the projection means isparallel with the image receiving means, the epipolar line matches acolumn or a row of a retinal plane. Thus, calculation becomes simple. Ifthe projection means is not parallel with the image receiving means, theparallel stereo system is configured through a calibration andrectification process. The calibration and rectification process can beeasily performed by those skilled in the art.

In a typical structural light scheme, a 3D image is restored by emittinga series of patterns to the object at a projection means and analyzingan obtained image at the image receiving means. In this case, it isnecessary to calculate an exact pixel correspondence relationshipbetween the projection means and the image receiving means. In an actualcase, however, signals transmitted by the projection means are mixed dueto various sources of error and received by the image receiving means.

The geometrical structure of the surface of the object 100 can accountfor such signal mixing. FIG. 2 shows a signal mixture model according tothe geometrical structure of the object surface. Even when the surfaceof the object 100 is flat and parallel with retinal planes 200 and 300of transmitting and receiving sides as in FIG. 2(a), signal mixingoccurs due to a difference in size and position between light projectedby the projection means and a receiving area of the image receivingmeans. If the surface of the object 100 is slanted toward the projectionmeans (Out-Slant) as in FIG. 2(b), light from adjacent pixels on theretinal plane 200 of the projection means may overlap and be incident ona corresponding pixel on the retinal plane 300 of the image receivingmeans. On the other hand, if the surface of the object 100 issignificantly slanted toward the retinal plane 200 of the projectionmeans (In-Slant) as in FIG. 2(c), some light from adjacent pixels on theretinal plane 200 of the projection means may be incident on acorresponding pixel on the retinal plane 300 of the image receivingmeans.

Further, when there is a separate object around the object 100(Transposition) as in FIG. 2(d), code transposition occurs on anadjacent surface and two remote separated surfaces of the separateobject. In the case of FIG. 2(e) (Out-Discontinuity), a code is deletedby a receiving area on the retinal plane 300 of the image receivingmeans and a shadow at the projection means. In the case of FIG. 2(f)(In-Discontinuity), when there is a rapid discontinuous point of thedepth at a boundary between cut surfaces of objects, light from adjacentpixels and remote pixels of the image receiving means may overlap and beincident on a corresponding pixel.

In addition to such signal mixing, the signal received by the imagereceiving means may be distorted due to system/environment noise such assurface reflectance variation, light scattering on the object surface,and ambient illumination variation. Surface reflectance variationchanges signal intensity, and light scattering on the object surfacecauses blurring or signal mixing. In addition, ambient illuminationvariation may cause signal intensity variation. Such factors such as thesignal mixing make a pixel point correspondence relationship between theprojection means and the image receiving means inexact. This inexactnessof the correspondence point relationship causes a fundamental error inobtaining a 3D depth image in a structural light scheme.

The signal separating and coding method according to the presentinvention is designed to exactly restore a pixel correspondence pointaddress of a projection means side through a mixed-signal separationprocess. A fundamental difference with the conventional coding method isthat the signal separating and coding method introduces a process ofseparating the pattern signal received by the image receiving means. Inorder to easily separate the signal, multiple light patterns areradiated in the encoding process from the projection means.

Signal mixing and the separation process in the present invention can bemodeled as follows:

(1) Signal mixing: when a signal is mixed with a neighboring signal,pixel brightness at the projection means side is assumed to be theweighted sum of the brightness of a pixel and the brightness of aneighboring pixel. That is, the relationship between a projection meansside signal and a image receiving means side signal can be representedas a linear mixed signal model, Y=XW. X R^((f×m)) and Y R^((f×m)) are amatrix consisting of a transmitting code x_(i)=x₁, . . . , x_(f) ^(T)sent by the projection means and a matrix consisting of a mixed imagesignal y₁=y₁, . . . , y_(f) ^(T) at the image receiving means side,respectively, and W R^(f×m) is a mixing matrix consisting of mixingcoefficients, where f denotes the length of the signal and m denotes thenumber of pixels. For example, when the temporal coding method is used,f indicates the number of frames.

(2) Signal separation: if X is a regular orthogonal matrix, the mixingmatrix is easily calculated as W=X^(T)Y. Accordingly, if one value isgreater than a neighboring signal value even when the signals are mixedupon transmitting patterns using the orthogonal signal, the index valueof the greatest code projected to orthogonal codes can be determined ascode y_(j)=argmax(j)W_(ij).

This model is a generalized model that does not consider the geometricalrelationship between the projection means and the image receiving means,in which a search for pixel correspondence between the projection meansand the image receiving means is performed on only the epipolar linerather than all images, similarly to stereo vision. In particular, thesearch may be performed based on the column or row of the image sincethe epipolar line matches a column or row of the projection means andthe image receiving means in the parallel stereo condition as describedabove. For example, if the means are arranged in parallel with eachother, the epipolar line matches these columns.

In the present invention, 3D depth imaging is performed by usingencoding, decoding and error-correcting methods using such a signalseparation concept.

First Embodiment

FIG. 3 is a flowchart illustrating a 3D imaging method using signalseparation according to an embodiment of the present invention.

As shown in FIG. 3, a unique transmitting side address is assigned to asignal corresponding to each pixel of a measurement target object, instep S1. In step S2, multiple light patterns are projected by theprojection means to encode a signal including the transmitting sideaddress. Each pattern is received in step S3. In step S4, an originalsignal is separated through a signal separation process. Acorrespondence point is calculated from the restored signal in step S5.

The steps are performed by a processing means that processes collecteddata. That is, the processing means used in the present invention is aprocessor that can perform steps S1 to S5. The processing means istypically implemented by a computer system.

An example of the signal encoding and decoding processes and the errordetection and correction used in the present invention will now bedescribed.

Signal Encoding

In the present invention, a proper signal coding method is needed tocalculate a pixel correspondence point using signal separation. Aconcrete example of the coding method allowing signal separationincludes projecting a plurality of patterns onto a time axis. Inparticular, an encoding method using adjacent orthogonal signals on atime axis may be used. A method of hierarchically arranging and usingorthogonal signals may be introduced to reduce the number of frames.

The structural light system used in the present invention can beregarded as one communication system having an information source, aninformation source encoding unit, a channel, a noise source, a decodingunit, and a receiving unit. When the projection means transmits encodedaddresses, distortion of the signal due to a geometrical structure of anobject and system/environment may be regarded as addition of the noiseto the channel. When the image receiving means receives signalscontaining noise, an original signal can be restored from the receivedsignals. If an order of signals sent by the projection means isobtained, a disparity with the received signal can be calculated and,accordingly, a 3D image can be restored.

In the structural light system using orthogonal signals according to anexemplary embodiment of the present invention, a communication processincludes an encoding process and a decoding process. In the encodingprocess, a unique address is assigned to a pixel position of each imageat a projection means side. In this case, the structural light systemsuffices to assign the unique address only to columns or rows of theimage instead of assigning a unique address to all images, because itcan consider an epipolar line geometrical condition, similarly to stereovision. When it is assumed that N pixels are placed on one epipolarline, the transmitting signal from the projection means includes Naddresses.S={s₁, s₂, . . . , S_(N)} N.

A binary code may be used to provide channel coding that is robustagainst environmental noise. That is, binary code, B={b₁, b₂, . . . ,b_(N)} corresponding to a source can be defined. If a completeorthogonal binary code (i.e., <b_(i), b_(j)>=0, i≠j) is used, the numberof images that should be obtained at the image receiving means needs tobe equal to the code length. Thus, while much calculation time isconsumed, an exact depth image can be obtained. In this manner, in thepresent invention, hierarchical orthogonal coding (HOC) in whichorthogonal signals are hierarchically arranged may be used to solve theproblem of an increasing number of frames. This HOC scheme is focused onshortening the length of the signal while keeping the nature of theorthogonal codes as much as possible.

FIG. 4 shows an example of this hierarchical orthogonal code. As shownin FIG. 4, all signals are classified into one or more layers, and eachlayer consists of a group of orthogonal signals. In the encodingprocess, a signal having N lengths is divided into L layers and eachlayer uses H orthogonal signals. While the entire signals are notorthogonal to one another, they are orthogonal to one another within auniquely designated range on an upper layer in a certain area of eachlayer. For example, if HOC uses four layers (L=4) and four orthogonalsignals (H₁=H₂=H₃=H₄=4) are used in each layer, the number of all thesignals becomes 256 (H₁×H₂×H₃×H₄=4⁴=256) and the code length becomes 16(H₁+H₂+H₃+H₄=16). That is, the image receiving means needs an image of16 frames to restore the address.

Signal Separation

In the signal separating and coding method according to an embodiment ofthe present invention, the signal separation process separates anoriginal signal from a mixed signal to obtain address values receivedfrom the projection means. Let the i-th position of a temporal andspatial image of f frames obtained in the image receiving means berepresented as I(i, t), and a brightness value of a pixel at thatposition as a vector y_(i)=y₁, . . . , y_(f). If HOC has L layers, avector y may be represented by one extension vector y_(i)=b₁, b₂, . . ., b_(L) ^(T) consisting of image vectors that represent a brightnessvalue in each layer. A suffix of the image vector b_(j) indicates thej-th layer. Because each layer uses H orthogonal signals, the vector bis represented by a linear combination of orthogonal signals, i.e.,b=Xc, where, c and X denote a coefficient vector and an orthogonal codematrix, respectively. That is, the vector b indicating the brightnessvalue at a specific position includes other signals affected by thegeometrical nature of an object and an environment and by reflectionfrom the object surface.

On the L-th layer, a coefficient vector c(i) at the i-th position can becalculated by the dot product of a transposed orthogonal matrix and thesignal vector b(i) measured at the position, i.e., C(i)=X^(T)b(i). Theextension coefficient vector c=c₁, c₂, . . . , c_(L) ^(T) correspondingto an extension vector y_(i)=b₁, b₂, . . . , b_(L) ^(T) for all thelayers can be obtained through this process. According to the HOChierarchical structure, a total number of possible addresses at the i-thposition becomes H^(L). That is, when four layers and four orthogonalsignals are used, a total of 256 possible candidates exist.

A conceptual diagram of such a signal separation process is shown inFIG. 5. In FIG. 5(a), 16 frames are obtained at the i-th position on theepipolar line of the camera image. The original signal is separatedthrough the dot product for the orthogonal matrix in FIG. 5(b), apossible code is selected in FIG. 5(c), and an exact address isdetermined in FIG. 5(d). Here, the process of determining the exactaddress is simplified by using typical signal intensity as a reference.Normally, the address is determined by a signal having maximum signalintensity.

After the address is determined, a depth image may be measured by simplegeometry. A method for measuring a depth using the original pixelposition and the received pixel position is well known in the art towhich the present invention pertains and thus a detailed description ofthe method will be omitted.

Error Detection and Correction

Because the received signal is a mixture of a plurality of signals, itcannot be said that the received signal always provides an exact addressjust because it has the maximum intensity. Accordingly, it is possibleto correct more errors using intensity at a neighboring position andstatistical distribution of restored candidate address values, as wellas the signal intensity, in determining an exact address from possiblecandidates. Such an error correction process can make the coding robustagainst a variety of noise.

The error correction method in the present invention includes errorcorrection using a rule, error correction using an evaluation function,and a mixture of conventional coding and the orthogonal coding.

(1) Error Correction Using HOC-Code Transposition Rule

This type of error correction uses the fact that restored address valuesexhibit a systematic pattern. Since a series of address values restoredon the epipolar line reflect the geometrical structure of an object andan environment, the address values exhibit the systematic pattern.Candidate addresses obtained through the signal separation process arechanged in the same pattern. It is possible to use an error correctionmethod that provides exact disparity based on this principle. Withaddress candidates of all positions, it is possible to detect andcorrect inexact addresses.

Groups of the candidate codes or the DMA pixel addresses {D_(i−k),p_(i−k)}, . . . , {D_(i−1), p_(i−1)}, {D_(i), p_(i)}, {D_(i−1),p_(i+1)}, . . . , {D_(i+k), p_(i+k)} are obtained along the epipolarline through the signal separation process for camera pixels C_(i−k), .. . , C_(i−1), C_(i+1), . . . , C_(i+k). A 16-bit HOC code is assigned apriority index p based on a relative signal intensity level calculatedin the signal separation process.

Accordingly, the code transposition rule for error detection is asfollows:

Plane rule: groups {D_(i−k), p_(i−k)}, . . . , {D_(i−1), p_(i−1)},{D_(i), p_(i)},{D_(i+1), p_(i+1)}, . . . , {D_(i−k), p_(i+k)} have thesame number of elements. Continuous change of pixel addresses is shownwhile in movement from C_(i−k) to C_(i+k).

Slanted surface rule: the number of the groups {D_(i−k), p_(i−k)}, . . ., {D_(i−1), p_(i−1)}, {D_(i), p_(i)}, {D_(i+1), p_(i+1)}, . . . ,{D_(i+k), p_(i+k)} gradually increases or decreases. The gradualincrease or decrease is treated as the same address is repeated byexpansion of a light projected onto the slanted surface, or continuousaddress overlap occurs by the light projected onto the slanted surface.Generally, the continuous change of the pixel address remains unchangedeven when an element increases or decreases.

Occluding rule: groups {D_(i−k), p_(i−k)}, . . . , {D_(i−1), p_(i−1)},{D_(i), p_(i)}, {D_(i+1), p_(i+1)}, . . . , {D_(i+k), p_(i+k)} indicatea rapid address change of the pixel whose address is deleted.

Shadow/shading rule: when groups {D_(i−k), p_(i−k)}, . . . , {D_(i−1),p_(i−1)}, {D_(i), p_(i)}, {D_(i+1), p_(i+1)}, . . . , {D_(i|k), p_(i k)}enter or exit the shadow, the number of elements gradually decreases tozero or gradually increases from zero.

(2) Error Correction Using Evaluation Function

The image measuring process according to the present invention is acomplex process related to selecting a possible code group and anaddress having the highest reliability from the group. A reliabilityindex function may be used in this process.

When HOC uses L layers and H orthogonal codes, a total number of addressvalue candidates at a specific position becomes L^(H). A more simplifiedmethod may be applied even when all cases are allowed to be considered.It can be assumed that one signal is created by mixing at most twodominant signals in order to reduce calculation complexity. The use ofthis assumption allows consideration of only 2^(L) candidates, not L^(H)candidates.

In other words, first, two representative signals are selected based onsignal intensity in each layer of HOC. Second, a reliability index canbe defined using, as factors, the size of a signal at a relevantposition, uncertainty of a difference with neighboring signals, andcontinuity reflecting a structural relationship of the object and theenvironment, in order to determine an address having the highestreliability. That is, a code y_(i) in the image at the y_(i) positionmay be determined by uncertainty reflecting a difference between thesignal size of y_(i) and the size of a neighboring signal, continuity ofa calculated depth at an adjacent position, and the like. That is, acost function h can be defined in Equation 1: $\begin{matrix}{{h\left( c_{i} \right)} = {{\overset{l}{\coprod\limits_{j = 1}}{{C_{i}(j)}{w_{i}(j)}}} + {\lambda\quad 1{\sum\left( {w_{i}{\log\left( \frac{1}{w_{i}} \right)}} \right)}} + {\lambda\quad 2{g\left( {c_{i},c_{k}} \right)}}}} & {{Equation}\quad 1}\end{matrix}$where, the weight vector y_(i), w_(i)=w₁, . . . , w_(f) ^(T), and thebinary code c_(i)=code y_(i).

In Equation 1, Ci(j) and w_(i)=(w₁, . . . , w_(f))^(T) denote a decodedsignal and a weight vector, respectively. The terms ΠC_(i)(j)w_(i)(j)and Σ(w_(i) log(1/w_(i))) correspond to the signal size of theorthogonal code Ci and a measured entropy value for uncertaintymeasuring, respectively. The function g(C_(i), C_(k)) indicates ameasured value corresponding to the geometrical continuity at a positionof the depth information. The function may be variously defined. Forexample, a simple slope may be used to define the function. λ1 and λ2are coefficients for adjusting the size of each factor.

(3) Error Correction Using Mixture Method

Temporal coding such as a conventional gray code may be mixed with theorthogonal coding to reduce the number of images and amount ofcalculation. The orthogonal code is then used to correct a temporalcoding error. That is, the orthogonal code can determine pixel positionmore accurately than the time code. Because the time code has a higheruncertainty at its lower bit, an error in the local window area can becorrected using the orthogonal code.

An orthogonal code having a length of K (e.g., three frames) is attachedbefore and after a typical temporal code. Since the orthogonal code usesonly a signal having a short length, its position in the entire imagecannot be completely determined. However, the position can be determinedwithin an adjacent local area. That is, the position can be selectedfrom K−1, K and K+1. This nature can be used for calibration to a localposition specified by the orthogonal code when the position restoredusing the orthogonal code does not match the position obtained using thetime code.

While the present invention has been described in connection with codingusing the orthogonal code and the hierarchical orthogonal code, it isnot limited to such a code. A method for restoring the original signalusing a variety of codes such as a pseudo orthogonal code, a mixture ofan orthogonal code and a non-orthogonal code, and a statisticallyindependent signal is possible.

The signal separating and coding method according to the presentinvention may be used in cooperation with a conventional coding method.That is, the signal separating and coding method of the presentinvention may be used to correct the restored address value withoutusing signal separation after signal separating and coding is performedon some of multiple light patterns.

Second Embodiment

The signal separation coding scheme according to the first embodiment isto estimate signal mixture between the projector pixel and the camerapixel in order to determine the pixel correspondence. In themixed-signal separation process, a result of projecting the receivedsignal to a hierarchical orthogonal signal group is represented in aprojector-camera signal space. 3D depth image calculation is a processof determining a projector side signal corresponding to each pixel ofthe camera in the projector-camera signal space.

The second embodiment suggests (1) a scheme for creating an image at atime point of a projector without explicit 3D restoration, (2) a newmethod for restoring 3D by simultaneously calculating a pixel-by-pixelcorrespondence between the created stereo image and the structurallight, using Hermann von Helmholtz's complementarity principle thatlight from one light source reflected on an object surface can beequally interpreted as light from a receiving side reflected on theobject surface and arriving at the light source, and using the signalseparation coding scheme.

Conventional dual photography has used a time-consuming analysis methodto analyze a geometrical relationship among the projector, the objectand the camera. The present invention uses a two-step analyzing methoddifferent from the conventional method. First, an encoded signal isgenerated to have hierarchical orthogonality considering epipolarconstraints in stereo vision, and is emitted through the projector.

The mixed signal reflected from the object and received by the camera isseparated by a decoding scheme for separating an original signal, and ascheme of efficiently estimating a conversion relationship between theprojector and the camera based on the intensity of the separated signalis applied.

Through this process, the image creation at a time point of theprojector and the estimation of the 3D depth image are simultaneouslyperformed by generating the image at a time point of the projector andcombining the image with the depth image in the structural light.

That is, the conventional method has been limitedly performed on anobject having simpler characteristics, such as reflection, transmissionand absorption, mainly in a restricted environment such as a studio. Onemain reason is that 3D image and calculation model creation is madeusing only interpretation of physical properties of the light source,the object and the sensor (e.g., a shape from shading, photometricstereo, etc.), and the geometrical relationship (e.g., passive or activestereo, structure from motion, etc.) therebetween.

The second embodiment according to an embodiment of the presentinvention overcomes the limitations of the conventional method bysuggesting a method for combining (1) a radiometry interpretationincluding scattering in the light source and at the object surface,reflection, and reception at the image sensor, and (2) a geometricalinterpretation including projection of a 2D image surface into 3D spaceat the projector, epipolar constraints among the projector, the objectand the camera, and projection from 3D space to the camera 2D imagesurface.

Accordingly, in the second embodiment, it is possible to obtain a newimage viewed from the projector side as shown in FIG. 9 in estimating asignal of the corresponding camera side at each position at theprojector side. In FIG. 9, the areas that are not restored at upper andlower ends are caused by the projector view and camera view that do notoverlap, and the noise is caused by the use of only the greatest valueof the mixed signal in calculation.

FIGS. 6 and 7 illustrate a signal separation map obtained by performingsignal separation on a specific epipolar line and a dual photographyimage created using the signal separation map.

FIG. 6 shows the relationship between separated signals, as an exampleof the relationship between a group of signals sent by the projector(horizontal axis) and a group of signals received by the camera(vertical axis). In FIG. 6, the brightness indicates the intensity ofcorrespondence. A small expanded area shows that correspondence betweenthe projector signal and the camera signal is M:N, not 1:1. The simplestmethod for determining signal correspondence between the projector andthe camera is to select a projector signal having the greatest intensityfor each address at the camera pixel side.

FIG. 7 shows a principle of creating a new image in a projector sideusing this signal relationship. Unlike FIG. 6, when all projectorsdetermine the address at the camera side, it is allowed to obtain a newimage viewed from the projector (a right side in FIG. 7). That is, abrightness value at an x, y position of the new obtained image can berepresented by dual image(x, y)=ref(arg max (j) Ty(j, x), y). refindicates an image at the projector side, and Ty indicates a mixing andseparating map at the y-th row (epipolar line).

FIGS. 8 and 9 are examples of dual photography using signal separationcoding.

That is, FIG. 8 shows a gray scale image in a refrigerator that isimaged at a camera side, and FIG. 9 shows an image at a new visualpoint, viewed from the projector side, which is created using aseparated signal on all epipolar lines. The camera and the projector areplaced at upper and lower positions when imaging an object,respectively. Accordingly, it can be observed that an air ventilationhole at a rear is located at a lower position when viewed from theprojector side rather than from the camera side.

That is, according to the second embodiment in which the interpretationof a physical aspect and the interpretation of a geometrical aspect arecombined, it is possible to partially solve difficulty in obtaining the3D image and creating the model due to scattering, reflection,absorption, and transmission on various object surfaces that may beeasily overlooked when only the geometrical aspect is considered. It isalso possible to solve the problem of numerous calculations makingimplementation using a sensing system difficult, in interpretingphysical conversion among the artificial light source, the object andthe camera when only the interpretation at the physical aspect isconsidered.

As described above, with the structural light-based 3D depth imagingmethod and system according to the present invention, it is possible toexactly separate the signal even when the signal received by the imagereceiving means undergoes overlapping and distortion due to thegeometrical structure of the object. It is also possible to obtain adepth image that is robust against ambient environmental noise.

With the 3D depth imaging method and system according to the presentinvention, it is also possible to solve the problem of numerouscalculations arising when the projector side creates the image. It isalso possible to obtain a 3D image that is robust against noise usingboth the projector image and the restored 3D geometrical information.

Exemplary embodiments of the present invention have been disclosedherein and, although specific terms are employed, they are used and areto be interpreted in a generic and descriptive sense only and not forpurpose of limitation. Accordingly, it will be understood by those ofordinary skill in the art that various changes in form and details maybe made without departing from the spirit and scope of the presentinvention as set forth in the following claims.

1. A method of 3D depth imaging by radiating light at a measurementtarget object using a projection means and imaging the light using animage receiving means, the method comprising the steps of: assigning aunique transmitting side address to a signal corresponding to each pixelof the projection means to encode the signal; projecting multiple lightpatterns at the projection means to transmit the signal; receiving theencoded signal at the image receiving means; separating the receivedsignal to restore the address; and determining a pixel position of theobject using the transmitting side address and the restored address. 2.The method according to claim 1, wherein the step of separating thereceived signal comprises an error correction step including the stepsof calculating possible address candidate values corresponding to thetransmitting side address; and determining an address according to arule for obtaining an exact address from the candidate values, and therule is selected from: a rule that the restored address values graduallyincrease or decrease with no continuity when the signal is obtained froma slanted surface of the object; a rule that the restored address valuesvanish or are transposed in part with no continuity when the signal isobtained from a cut surface of the object; and a rule that the restoredaddress values vanish with no continuity when the signal vanishes by ashadow of the object.
 3. The method according to claim 1, wherein thestep of separating the received signal comprises an error correctionstep including the steps of: calculating possible candidate values of anaddress corresponding to the transmitting side address; and defining anevaluation function by considering a group of address candidates atone's position and address candidates at neighboring positions among thecandidates, and determining the address using a searching method formaximizing or minimizing the evaluation function.
 4. The methodaccording to claim 1, wherein the step of separating the received signalcomprises an error correction step including the steps of: calculatingpossible candidate values of an address corresponding to thetransmitting side address; and calculating an address value at aposition having high reliability, fixing the address value at theposition, and then determining the candidates from address values inother areas in order to select the address from a plurality of candidategroups consisting of the candidates.
 5. The method according to any oneof claims 2 to 4, wherein the light pattern is a binary light pattern, acolor light pattern, a light pattern having a brightness value of a grayscale, a light pattern of an ultraviolet or infrared area in anon-visible area, or a mixture of these.
 6. The method according to anyone of claims 2 to 4, wherein the step of encoding the signal comprisesusing orthogonal signals among adjacent signals at the projection means.7. The method according to claim 6, wherein all the orthogonal signalsare classified into one or more hierarchical signals, and use codes thatare orthogonal to one another in each layer.
 8. The method according toclaim 6, wherein all the orthogonal signals are classified into one ormore hierarchical signals, and use a mixture of codes that areorthogonal to one another in each layer and codes that are notorthogonal to one another.
 9. The method according to any one of claims2 to 4, wherein the step of encoding the signal comprises using pseudoorthogonal signals having orthogonality among adjacent signals of theprojection means.
 10. The method according to any one of claims 2 to 4,wherein the step of encoding the signal comprises using statisticallyindependent signals among adjacent signals of the projection means. 11.A method of 3D depth imaging in a system including a projector and acamera, the method comprising the steps of: emitting a pattern from theprojector to an object; receiving the pattern using the camera;calculating a pixel correspondence relationship between the projectorand the camera through signal separation on each epipolar line; andcreating a new image at a time point of the projector after thecalculating step is performed.
 12. A method for restoring a stereo imagein a system including a projector and a camera, the method comprisingthe steps of: emitting a pattern from the projector to an object;receiving the pattern using the camera; calculating a pixelcorrespondence relationship between the projector and the camera throughsignal separation on each epipolar line; and creating a new image at atime point of the projector after the calculating step is performed, inwhich the stereo image is restored using an image obtained by the cameraand the new image created by the projector.
 13. A method for restoring ameasuring depth in a system including a projector and a camera, whereinthe method restores the measuring depth using both a structurallight-based 3D depth imaging method comprising the steps: assigning aunique transmitting side address to a signal corresponding to each pixelof the camera to encode the signal; projecting multiple patterns fromthe projector to transmit the signal; receiving the encoded signal atthe camera; separating the received signal to restore the address; anddetermining a pixel position of an object using the transmitting sideaddress and the restored address, and a stereo image restoring methodcomprising the steps of: emitting a pattern from the projector to theobject; receiving the pattern using the camera; calculating a pixelcorrespondence relationship between the projector and the camera throughsignal separation on each epipolar line; and creating a new image at atime point of the projector after the calculating step is performed. 14.A structural light-based 3D depth imaging system comprising: projectionmeans for radiating light at a measurement target object; imagereceiving means for imaging the light radiated from the projectionmeans; and processing means for imaging a 3D depth of the measurementtarget object, the processing means sequentially performing: encoding asignal corresponding to each pixel of the projection means by assigninga unique transmitting side address to the signal; projecting multiplelight patterns at the projection means to transmit the signal; receivingthe encoded signal at the image receiving means; separating the receivedsignal to restore an address; and determining a pixel position of theobject using the transmitting side address and the restored address. 15.The system according to claim 14, wherein the projection means is aprojector, and the image receiving means is a camera.
 16. The methodaccording to claim 1, wherein the projection means is a projector, andthe image receiving means is a camera.